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Abstract - This paper describes a general overview of applying 
Independent Verification and Validation (IV & V) to Automatic Test 
Equipment (A TE). The overview Is not inclusive of all IV& V 
activities that can occur or of all development and maintenance 
Items that can be validated and verified, during the IV&V 
process. A sampling of possible IV&V activities that can occur 
within each phase of the ATE life cycle are described. 


I. INTRODUCTION 

Automatic Test Equipment (ATE) is becoming 
increasingly more sophisticated and complex, with an 
even greater dependency on software. With the onset 
of Versa Modular Eurocard (VME) Extensions for 
Instrumentation (VXI) technology into ATE, which is 
supposed to bring about the promise of interchangeable 
instrument components, ATE customers are forced to 
contend with system integration and software issues. 
One way the ATE customer can combat these problems 
is to have a separate Independent Verification and 
Validation (IV&V) organization employ rigorous 
methodologies to evaluate the correctness and quality 
of the ATE product throughout its life cycle. IV&V is a 
systems engineering process where verification 
determines if the ATE meets its specifications, and 
validation determines if the ATE performs to the 
customer’s expectations. 

IV&V has the highest potential payoff of assuring a safe 
and reliable system if it is initiated at the beginning of 
the acquisition life cycle and continued throughout the 
acceptance of the system. Fig. 1 demonstrates how 
studies have shown that IV&V effects are more 
pronounced when IV&V activities begin early in the 
software development life cycle, but later application of 
IV&V is still deemed to have a significant impact. 

IV&V is an effective technique to reducing costs, 
schedule, and performance risks on the development of 
complex ATE, and a “tool” to efficiently and effectively 
manage ATE development risks. The IV&V 
organization has the ability to perform a focused and 
systematic technical evaluation of hardware and 


software processes and products. When performed in 
parallel with the ATE development life cycle, IV&V 
provides for the early detection and identification of risk 
elements. The program is then able to take action to 
mitigate these risks early in the life cycle. 



Fig. 1 IV&V Error Detection and Associated Cost (taken from 
briefing by James Arthur (Virginia Tech/LaRC), 10 Oct 96). 


II. IV&V TEAM MEMBERS 

The IV&V team should consist of members from the 
ATE developer and program management sites, and the 
IV&V organization. The paradigm of IV&V is to locate 
analysts at the IV&V site, and the “eyes, ears, and 
position advocates" of IV&V at the ATE development 
and program management sites where physical 
presence is necessary for the exchange of information. 
The customer provides program management focus, 
while the developer provides for inside development 
requirement focus. With the developer, problems are 




worked at the lowest level with the aim to resolve 
problems at earliest possible moment, thus requiring a 
small impact to correct. 

III. IV&V ACTIVITIES PRIOR TO LIFE CYCLE 

Starting IV&V activities prior to the acquisition phase 
can lend itself optimum benefits. One large benefit 
would be the selection of a reliable, highly qualified 
organization to develop the ATE. The selection of the 
ATE development organization can be defined by the 
criticality of the ATE to be developed and the end cost. 
The following guidelines can be applied during the 
selection process: 

• Develop a specification for function and 
performance of the desired ATE, along with defined 
measurable characteristics whenever possible. 

• Select three or four candidate ATE that best meet 
the specification. 

• Develop a comparison matrix that represents a 
head-to-head comparison of key functions. 

• Evaluate each ATE based on past product quality, 
vendor support, product direction, reputation, etc. 

• Contact current users of the ATE and ask for 
opinions. 

In the final analysis, the selection of the ATE 
development organization should be based on the 
outcome of this process. 

IV. IV&V LIFE CYCLE ACTIVITIES 

The following sections describe the IV&V activities that 
may be conducted during the acquisition, development, 
and maintenance of ATE. These IV&V activities may 
include the analysis, evaluation, review, inspection, 
assessment, and testing of whether the hardware and 
software of the ATE meet its system requirements. 

A. Acquisition 

During the acquisition phase the IV&V organization will 
assist program management in scoping the effort of 
IV&V required on the development project. The scope 
is determined so as to minimize the risk within the 
customer’s budget. This is done by determining the 
objective of the IV&V (e.g. - technical complexity, safety, 
mission critical, security), the criticality and importance 
of the ATE, and the IV&V budget, including test facilities 
and tools as required. Once these steps are completed, 
the IV&V effort is tailored to fit the ATE development 


program’s need and a schedule for each IV&V task is 
defined along with the IV&V project plan. 

Other important activities will include verifying that the 
contract specifies the correct program deliverables and 
schedule including definition of ownership, warranty, 
and copyright, and that the acceptance criteria and 
procedures for each deliverable are clearly, correctly, 
and completely specified. These activities can help 
ensure that when the ATE is delivered, items such as 
Prime Item Development Specifications are usable, and 
instrumentation operation capabilities met these 
specifications. A big problem in the past with ATE has 
been instruments that did not meet the performance 
requirements. For example, a test requirement calls for 
a 30-volt, 10-amp signal, and the power supplies in the 
ATE only supply up to a 20-volt, 5-amp signal. Or a 
10-volt sine wave is required, and the function generator 
in the ATE only supplies a 9.23-volt sine wave. 

B. Development 

As illustrated in Fig. 2, ATE usually consists a 
conglomerate of four basic elements: controller, 
instruments, switching, and interface. 



Fig. 2 ATE four basic elements. 


The five key ATE software elements include the test 
program (source and object code), compiler, operating 
system, and the test executive as shown in Fig. 3. 



Fig. 3 Five key ATE software elements. 


During ATE development, a continuous review of 
associated documentation is performed as part of the 
IV&V effort, and the software/system engineering 





processes are closely monitored for any weak areas. 
Technical reviews and audits are assessed for the 
acceptability of the products under review and the extent 
to which milestones and life cycle goals are met. 
Assessing the reviews and audits increases program 
visibility and provides prioritized recommendations for 
corrective actions to maintain a focus on those 
problems posing the highest risk. Risk identification 
occurs throughout ATE development, identifying current 
and probable problem areas in the ATE development, 
and quantifying these risks associated with the problem 
areas, recommending alternatives to reduce the risks. 
Problem areas may consist of project risks (potential 
budgetary, schedule, personnel, resource, customer, 
and requirements problems and their impact on the 
development project) and technical risks (potential 
design, implementation, interfacing, verification, and 
maintenance problems). 

1.) Requirements 

During the requirements phase, the following IV&V 
activities ensure the correctness, completeness, 
accuracy, testability, and consistency of the 
performance and quality requirements: 

System requirements traceability analysis - trace 
software and hardware requirements back to the system 
requirements in the Performance Specification and/or 
the Operational Concept Document. 

System requirements evaluation - evaluate the 
Hardware Development Specification, Software 
Requirements Specification (SRS), and Interface 
Requirements Specification for readability, correctness, 
consistency, completeness, accuracy, and testability. 

Interface analysis - Evaluate the Hardware 
Development Specification with controller, instruments, 
and interface panel requirements documentation, and 
the SRS with the hardware, user, operator, and software 
interface requirements documentation, for correctness, 
consistency, completeness, accuracy, and testability. 

Criticality analysis - Determine the criticality and 
importance of each hardware and software requirement 
(catastrophic, critical, marginal, negligible) 

Hazard analysis - Analyze system, hardware, software, 
and interface requirements to identify conditions, and 
factors which cause or contribute to a hazard, especially 
on ATE for nuclear testing. 

Configuration management assessment - Assess the 
completeness and adequacy of the configuration 
management process. 


2.) Design 

The following IV&V activities during the design phase 
determine if the design of the ATE is a correct, 
accurate, and complete transformation of the system, 
hardware, and software requirements: 

Design traceability analysis - Trace design document 
to requirements and development specifications, and 
vice versa for hardware and software. Analyze 
identified relationships for correctness, consistency, 
completeness, and accuracy. 

Design evaluation - For hardware, evaluate equipment 
layout drawings; block diagrams, schematics, and logic 
diagrams with interface control drawings; Mock-ups, 
models, breadboards, or prototype hardware when 
appropriate; mechanical, electrical, and packaging 
design of consoles, racks, drawers, printed circuit 
boards, connectors, and instruments; firmware to be 
provided with the system; electromagnetic compatibility; 
and configuration item development schedule. 

For software, evaluate functional flow; storage allocation 
data; control functions; computer software configuration 
item structure; development and test tools; security and 
safety; design, operation, and support documentation. 

Interface analysis - Evaluate the hardware design 
document with controller, instruments, and interface 
panel requirements documentation, and the software 
design document with the hardware, user, operator, and 
software interface requirements documentation, for 
correctness, consistency, completeness, accuracy, and 
testability. 

Test analysis - Verify design tests and criteria for 
component testing, integration testing, system testing, 
and acceptance testing. 

Hazard analysis - For hardware, verify current and 
voltage characteristics of the switches; current levels 
imposed by Unit Under Test failures; environmental 
control and thermal aspects; and power generation and 
grounding. For software verify that critical logic design 
and associated data elements correctly implements the 
critical requirements and introduces no new hazards. 

3.) Implementation 

The following IV&V activities during the implementation 
phase verify and validate the transformation of design 
into the actual build of the hardware and software: 

Source code traceability - Trace source code to 
corresponding design specification and vice versa. 



Analyze identified relationships for correctness and 
completeness. 

Source code and documentation evaluation - 

Evaluate source code for correctness, consistency, 
completeness, readability, testability, and accuracy. 
Evaluate source code for conformity with instituted 
standards, practices, and conventions. Validate source 
code against system requirements. Verify source code 
documentation (e.g. flowcharts, pseudo code, object 
diagrams) for completeness and consistency with 
source code. 

Hardware and software component test execution - 
Perform IV&V component testing as required by the 
component test procedures of the hardware and 
software. Analyze results to determine that software 
and hardware correctly implements the design. 

Interface analysis - For hardware, evaluate hardware 
with interface design documentation verifying that the 
test program/controller converts test program to 
machine code; controller and instruments transmits & 
receives instrument instructions and data; instruments 
and switching provides signal and signal routing to 
switch; and switching/interface panel mechanical and 
electrical interface conversion. 

For software, evaluate source code with hardware, 
operator, and software interface design documentation 
for correctness, consistency, completeness, and 
accuracy. 

Hazard analysis - Update Hazards Analysis as a result 
of code and hardware interface verification tasks. 

4.) Test 

For the duration of the test phase, the following IV&V 
activities ensure that the software, hardware, and 
system requirements are satisfied by the execution of 
qualification and system tests: 

Traceability analysis - Verify valid relationship between 
the requirements, test plan, test cases, and test 
procedures. 

Integration test execution - Perform integration testing 
in accordance with test procedures. Analyze results to 
determine if hardware and software implements 
requirements and design, and that software and 
hardware components function correctly together. 
Document any discrepancies between actual and 
expected results. 


System test execution - Perform system testing in 
accordance with test procedures. Analyze results to 
determine if hardware and software satisfies system 
objectives, and identify any hardware and software test 
limitations. 

Acceptance test execution - Perform acceptance 
testing in accordance with test procedures. Analyze 
results to determine if hardware and software satisfies 
system objectives. 

C. Maintenance 

The maintenance IV&V activities addresses any 
modifications, enhancements, or additions to the 
hardware or software after the ATE has been delivered 
to the customer. Enhancements may include replacing 
existing ATE computer operating system with a PC. 
Additions may include adding a VXI card and software 
drivers to the ATE. The maintenance IV&V activities 
reapplies the IV&V activities of the definition and 
development phases, but do so in the context of the 
existing hardware and software. 

V. COST OF IV&V 

Various sources have stated that the cost of IV&V can 
range from 10 - 50% of the ATE development cost. The 
cost will depend on the level of IV&V effort required. 
The degree or level of IV&V applied is determined by 
reliability and mission critical concerns and is 
proportionate to ATE development risk and function 
criticality. Criticality and development risks of each ATE 
function drives the amount of IV&V to be performed 
(e.g., detailed code analysis is performed only for more 
critical/high risk functions). How much IV&V versus 
criticality is then negotiated with the program 
management relative to availability of funding for IV&V. 

VI. BENEFITS 

Applying IV&V in all phases of the life cycle can bring 
about various benefits. IV&V can offer the following 
benefits to the development and maintenance of ATE: 

• Identify non-essential requirements which when 
eliminated reduce development, test, and 
maintenance cost. 

• Identify the lack of a critical functional definition 
required for the execution of the command and 
control software of the ATE. 

• Determine the actual performance of the ATE 
versus the performance specification when the 



instrumentation parameters are critical to the ATE 
operating capability. 

• Identify any risk associated with the interfaces of the 
VXI components. 

• Permit fewer latent defects, resulting in reduced 
effort and time spent during testing and 
maintenance. 

• Detect ambiguous or unclear specifications that can 
lead to the introduction of critical faults downstream 
in the development process. 

• Reduce the effort required to fix individual faults. 

• Provide higher ATE reliability resulting in greater 
customer satisfaction. 

• Identify reusable components (requirements 
document, architectures, code, testing components) 
that may be used in future developments. 

• Reduce maintenance costs. 

• Reduce overall life cycle cost of ATE. 

VII. CONCLUSION 

IV&V provides for increased visibility into the ATE 
development process and ensures the customer’s 
needs are adequately reflected in the work. While a 
structured set of standards, procedures, and 
requirements can greatly improve the chances of 
producing a good result, these alone can not possibly 


produce truly great products. While the IV&V 
organization should also review the standards and 
procedures, they must look beyond them to see if a 
first-class system engineering job is being performed 
and if key risks and feasibility issues are being 
addressed. 

To prevent system integration problems of advancing 
sophisticated ATE, IV&V can counteract these problems 
by establishing high confidence in state-of-the-art ATE 
reliability, allowing for an open architecture system that 
will easily allow for the insertion of new instrumentation 
and software. 
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